The Validation of Learning Systems { A Case

There is an obvious necessity to validate resp. verify complex systems. If human experts are involved in the implementation of any validation scenario, there arises the problem of the experts' competence. As a case study, the problem of expertise for systems validation is investigated in the area of learning systems validation. It turns out that certain human expertise suucient to accomplish certain validation tasks is substantially non-recursive. Consequently, there is no way to replace humans by computer programs for those validation tasks. There is an obvious necessity to validate resp. verify complex systems. It might easily happen that :::the inability to adequately evaluate systems may become the limiting factor in our ability to employ systems that our technology and knowledge will allow us to design. (cf. (Wise & Wise 1993)) Unfortunately, there are numerous severe accidents bearing abundant evidence for the truly urgent need for complex systems validation. Besides spectacular cases, daily experience with more or less invalid systems is providing paramount illustrative examples. Progress in the area of validation and veriication of complex systems requires both disciplinary results and solutions in the humanities including cognitive psychology, e.g. Even social and political aspects come into play. The authors refrain from an in-depth discussion. Following (Boehm 1984) and (O'Keefe & O'Leary 1993), validation is distinguished from veriication by the illustrative circumscription of dealing with building the right system, whereas veriication deals with building the system right. The focus of the present paper is on systems validation, which { according to the perspective cited above { is less constrained and less formalized than veriication. Assume any computer systems which are designed and implemented for an interactive use to assist some human beings in open loops of human{machine interactions of a usually unforeseeable duration in time. The validation task is substantially complicated, if it is intermediately undecidable whether or not some human{machine cooperation will eventually succeed. Nontrivial problems of inductive learning are quite typical representatives of such a collection of problems attacked through complex and usually time consuming sequences of human{machine interactions. Knowledge discovery in data bases, for instance, is an interesting application domain for those learning approaches. For assessing those systems' validity, there have been proposed validation scenarios of several types (cf. (Knauf, Philippow, & Gonzalez 1997), e.g.). As soon as human experts are involved in the implementation of validation scenarios, there arises the problem of their competence. An investigation of validation …